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the  Naval  Sea  Systems  Command  (NAVSEA-61R2).  The  objective  of  the  subproject  is  to 
provide  human  factors  engineering  support  to  the  development  of  Navy  management 
information  systems. 
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Research  and  Development  Center  who  served  as  users  in  this  study.  Especially  important 
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contractor,  in  developing  and  programming  the  study  materials. 


B  E.  BACON 
Captain,  U.S.  Navy 
Commanding  Officer 


3.  W.  TWEEDDALE 
Technical  Director 


SUMMARY 


Problem 

A  recent  General  Services  Administration  survey  (1985)  reported  that  the  number  of 
small  computers  bought  by  the  Department  of  Defense  increased  11-fold  in  1984  over 
1983.  Of  those  small  computers,  the  majority  were  IBM  PCs  and  Zeniths.  User 
complaints  about  the  adequacy  of  system  documentation  and  error  messages  of  these  disk- 
operating  systems  (DOS)  have  been  numerous.  This  study  was  done  to  assess  the  relative 
effectiveness  of  system  error  messages. 

Method 


A  sample  of  computer  system  error  messages  used  by  the  Wang,  Zenith,  and  IBM 
microcomputers1  was  interpreted  on-line  by  40  subjects  with  varying  computer  expertise, 
measured  by  an  experience  profile  administered  before  the  test. 

Findings 


1.  On  a  four-alternative,  forced-choice  test  (i.e.,  where  chance  performance  equals 
25  percent  correct),  subjects  correctly  interpreted  only  57  percent  of  the  error  messages. 

2.  Experienced  users  correctly  interpreted  significantly  more  error  messages  than 
inexperienced  users  even  though  the  contextual  cues  normally  available  to  them  were  not 
present. 


3.  Both  experienced  and  inexperienced  user  groups  spent  equal  time  on  the 
interpretation  tasks,  so  differences  in  performance  between  the  two  groups  cannot  be 
explained  by  time  spent  on  the  test. 

4.  Successful  interpretation  of  error  messages  by  users  was  more  a  function  of  the 
number  of  operating  systems  the  users  had  been  exposed  to  than  of  any  other  computer- 
related  experience. 

5.  Seventy-three  percent  of  the  error  messages  used  were  common  to  the  IBM  PC 
system.  Not  surprisingly,  the  IBM  users  demonstrated  superiority  over  non-IBM  users  in 
error  message  interpretation. 

Conclusions  and  Recommendations 


1.  Error  messages  need  to  be  more  understandable  for  both  experienced  and  inexpe¬ 
rienced  operators  in  order  for  them  to  use  their  systems  efficiently. 

2.  Most  importantly,  messages  should  be  specific  and  prescriptive.  Other  attributes 
of  effective  error  messages  include  prompts  that  use  familiar  vocabulary  and  present 
feedback  in  an  appropriate  format. 

3.  A  future  study  should  involve  the  redesign  of  messages  that  were  poorly 
understood.  Those  messages  that  already  yield  a  high  percentage  of  correct 
interpretations  could  serve  as  controls  for  measurement  of  test-retest  reliability. 
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INTRODUCTION 


A  recent  General  Services  Administration  survey  (1985)  reported  that  the  number  of 

C°^?U}uCS  DOUSht  bY  the  Department  of  Defense  increased  by  11-fold  in  1984  over 
1983.  Of  tnose  small  computers,  the  majority  were  IBM  PCs  and  Zeniths.  User 
complaints  about  the  adequacy  of  system  documentation  and  error  messages  of  these  disk- 
operating  systems  (DOS)  have  been  numerous.  The  wording  of  error  messages  and  prompts 
affects  the  ease  with  which  novices  learn  a  system  and  the  efficiency  with  which 
experienced  users  traverse  a  system. 


Several  dimensions  of  error  messages  and  other  forms  of  system  communications 
have  been  studied,  such  as  ways  to  name  objects  (Baggett,  1983),  ways  to  abbreviate 
(Ehrenreich,  1985),  word  counts  and  lengths,  and  readability  indices  (Kincaid  & 
Delionbach,  1973).  Implicit  in  the  learning  process  of  any  computer  system  is  the  belief 
that  experience  with  the  system  is  the  best  teacher  and  that  once  the  command  structure 
is  learned  it  will  not  be  forgotten.  Also  it  is  reasoned  that  experience  with  one  system 
wi  transfer  to  another  and  that  "computer  expertise"  is  a  global  skill  nurtured  by  the 
sampling  of  as  many  systems  as  possible. 


Developers  of  user-computer  interfaces  have  proposed  guidelines  for  the  design  of 
error  messages  which  are  intended  to  ensure  their  effectiveness.  Shneiderman  (1982)  has 
oeen  a  leading  proponent  and  recommends  that  messages  be  specific,  offer  constructive 
guidance,  be  user-oriented  in  style,  and  have  an  appropriate  physical  format.  Some 
system  designers  argue  that  the  context  in  which  the  message  appears  is  the  key  factor  in 
correct  interpretation.  For  example,  the  message  is  understandable  because  of  the 
particular  task  the  user  is  doing— a  system  operation,  programming,  or  software  applica¬ 
tion.  Their  argument  is  that  the  task  to  be  done  clarifies  what  the  message  means. 

Computer  experts  gain  their  skills  from  using  different  systems  to  accomplish  a 
variety  of  tasks.  The  present  study  is  aimed  at  determining  the  effect  of  generalized 
computer  experience  in  interpreting  context-free  system  error  messages  associated  with 
requently  encountered  tasks.  If  message  comprehension  is  a  function  of  the  user's 
learning  experiences  with  computer  systems,  then  computer  experts  should  do  signifi- 
V  e*ter  in  interpreting  system  error  messages  than  less  experienced  users.  But,  if 
feedback  is  presented  without  the  benefit  of  knowledge  about  the  task  to  be  accom- 
p  is  ed,  there  should  be  no  difference  between  experienced  and  inexperienced  users  In 
error  message  interpretation.  This  study  was  done  first  to  assess  how  understandable 
error  messages  are  and  secondly  to  compare  experienced  and  inexperienced  user  perfor¬ 
mance  in  interpreting  them.  v 


METHOD 


Test  Materials 


m , .  System  error  messages  representative  of  various  tasks  from  the  Wang,  Zenith,  and 
IBM  microcomputers  were  sampled  and  put  in  the  form  of  a  41-item  multiple-choice 
test.  For  each  item  there  was  one  correct  and  three  incorrect  responses.  Figure  1 
reflects  the  origins  of  test  items;  some  messages  were  common  to  all  systems?  The 

DOsfl-DOSr  aWn  fr°m  VadantS  °f  the  Microsoft  disl<  operating  system  (PC-DOS,  MS- 
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Figure  1.  Origins  of  test  items. 


Messages  were  presented  to  users  on  the  Zenith-120  monitor.  The  41  test  items 
appeared  in  a  different  random  order  for  each  subject.  After  the  examinee  received  on¬ 
line  instructions  about  the  nature  of  the  test,  items  were  presented  one  at  a  time  to  the 
subject,  who  was  kept  apprised  of  how  many  items  remained  as  an  indication  of  progress. 
Responses  could  not  be  changed  once  they  were  entered  on  the  keyboard. 

Error  message  test  items  represented  tasks  that  all  microcomputer  users  need  to 
perform  at  one  time  or  another.  Of  the  41  questions,  17  dealt  with  disk  operations,  11 
with  file  operations,  2  with  data  transmission,  1  each  with  the  arithmetic,  printer,  and 
editor  functions,  and  8  with  error  messages  that  could  apply  to  any  of  the  preceding 
operations,  for  example,  SYNTAX  ERROR.  A  copy  of  the  test  appears  in  Appendix  A. 

User  Profile 


Each  subject  completed  a  paper-and-pencil  questionnaire  about  personal  background 
(Appendix  B).  Information  concerned  gender,  age,  highest  education  level,  and  specifics 
related  to  computer  expertise.  The  computer-related  information  included  items  about 
number  of  academic  courses  completed  that  involved  work  in  introductory  data  processing 
(e.g.,  information  science,  computerized  data  analysis).  Data  were  also  gathered  on  the 
subject's  training  in  computer  languages  and  the  frequency  of  programming  in  each  of 
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these  languages  during  the  previous  month.  If  the  subject  used  a  computer  at  home  or  at 
work,  he  or  she  was  asked  to  indicate  the  purposes  for  which  it  was  used  and  the  number 
of  word  processing  text  editors  used.  Lastly,  the  types  of  interactive  systems  used  and 
the  operating  systems  experienced  were  recorded.  The  responses  to  the  questionnaire 
were  scored  for  each  subject;  maximum  score  possible  was  52  points. 

Allotment  of  points  possible  was  as  follows:  Academic  preparation  (9),  computer 
programming  experience  (18),  purposes  and  usage  (14),  number  of  interactive  systems  used 
(6),  and  operating  systems  used  (5). 

Subject  Sample 

A  total  of  40  people  participated  in  the  study.  The  group  was  made  up  of  28  males 
and  12  females,  both  Navy  personnel  and  civilians.  Ages  ranged  from  14  to  over  56  years. 
Education  ranged  from  junior  high  school  through  the  doctorate  level.  Experience  was 
wide-ranging,  from  subjects  who  had  no  specialized  computer  experience  to  others  with 
college  degrees  in  computer  science.  Among  the  40  subjects,  experience  with  nine 
programming  languages  and  thirteen  operating  systems  was  reported.  Use  of  computers 
spanned  nine  categories  (e.g.,  educational,  business  applications)  and  30  different  inter¬ 
active  computer  systems. 

Profile  scores  for  the  40  subjects  ranged  from  2  to  36,  with  a  mean  of  19.1  and  a 
standard  deviation  of  9.4.  Some  subjects  had  no  exposure  to  computers  while  others  had  a 
great  deal  of  experience  with  programming  languages  and  with  several  operating  systems. 

Hardware  and  Software 


Test  items  were  presented  to  the  subjects  and  response  data  were  automatically 
recorded  on  a  Zenith-120  microcomputer  equipped  with  704  K  bytes  memory,  a  5.25"  disk 
drive,  and  a  Winchester  hard  disk.  The  visual  display  terminal  on  which  error  messages 
were  presented  was  a  green  phosphorous  monitor;  it  measured  11  inches  diagonally  and 
was  built  into  the  computer. 

The  software  that  produced  the  error  messages  and  stored  the  subjects'  responses  was 
developed  in  Microsoft  Z-BASIC.  The  program's  structure  is  depicted  in  Figure  2. 

Data  Collection  and  Experimental  Design 

Of  primary  interest  in  this  study  were  the  users'  scores  on  the  41  items.  These 
results  were  recorded  on-line  and  analyzed  as  to  the  number  of  correct  responses.  Of 
secondary  interest  was  the  subject's  time  spent  on  the  test.  Total  time  was  recorded  on¬ 
line  and  represented  the  accumulation  of  times  elapsed  between  the  instant  the  examinee 
saw  the  message  on  the  screen  and  when  he  or  she  entered  a  response.  It  did  not  include 
pre-test  instructional  time,  screen  refresh  time,  or  end-of-test  instructions.  Subjects' 
profile  scores  were  recorded  off-line  as  a  measure  of  their  computer  expertise. 

The  analysis  consisted  of  calculating  overall  statistics  on  user  performance  such  as 
test  scores,  profile  scores,  and  test  time.  Experienced  and  inexperienced  users  were  also 
compared  on  these  variables.  In  addition  to  this  comparison  of  test  scores  and  test  times, 
qualitative  experience  with  various  microcomputers  and  operating  systems  was  also 
correlated  with  test  scores. 

The  40  users  were  divided  into  two  groups  of  20,  with  one  group  (the  inexperienced 
users)  having  user  profile  scores  below  the  median  (Md  =  16.0)  and  the  other  (experienced 
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Figure  2.  Flowchart  of  error  message  test  program. 
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users)  having  profile  scores  above  the  median.  A  subject  with  a  profile  score  just  below 
the  median  (e.g.,  14.0-16.0)  had  typically  completed  only  one  computer  course,  been 
exposed  to  one  programming  language,  used  a  computer  for  two  purposes,  used  only  one 
interactive  computer  system,  and  had  experience  with  only  one  operating  system.  The 
division  of  users  into  these  two  groups  produced  the  expected  significant  difference 
(p  <  .001)  in  experience  level  as  measured  by  the  User  Questionnaire.  This  difference  was 
useful  for  studying  the  effect  of  experience  on  error  message  interpretation. 


RESULTS 


Group  Performance 

The  mean  test  score  (number  of  correct  responses)  for  all  users  was  23.3,  with  a  high 
score  of  32  and  a  low  score  of  14;  the  standard  deviation  was  4.2.  Test  scores  varied  less 
than  profile  scores,  having  half  the  range  and  standard  deviation.  The  average  score  of 
most  achievement  tests  should  fall  about  midway  between  a  chance  score  and  a  score  of 
100  percent  (Marshall  &  Hales,  1972).  With  a  41-item  test,  chance  score  on  4-choice 
items  equals  25  percent,  or  10.25,  so  the  midscore  should  equal  (41.0  -10.25/2)  +  10.25  = 
25.6.  The  mean  test  score  of  23.3  was  thus  2.3  points  less  than  the  recommended  score. 
Marshall  and  Hales  (1972)  also  advise  that  the  mean  of  the  indices  of  item  difficulty 
should  fall  midway  between  chance  score  and  100  percent  (25  percent  -100  percent)  or 
62.5  percent.  The  mean  of  the  item  difficulty  indices  of  the  error  message  test  was 
23.3/41  or  56.S.  The  best  estimate  of  test  reliability  is  Saupe's  formula  R16K,  reported  by 
McMorris  (1972),  which  yields  a  reliability  coefficient  of  0.64.  A  corresponding  standard 
error  of  measurement  (SE  =  SD/  1  -  R)  resulted  in  2.52,  relatively  small  compared  to 
many  achievement  tests  (McMorris,  1972). 

Experienced  Versus  Inexperienced  Users 

After  the  degree  of  accuracy  was  determined  to  be  at  only  57  percent,  an  analysis 
was  done  on  the  effect  of  experience  in  interpreting  error  messages.  The  test  scores  of 
the  experienced  and  inexperienced  groups  were  compared  and  are  displayed  in  Table  1. 


Table  1 

Comparison  of  Inexperienced  and  Experienced 
Group  Test  Scores 


Inexperienced 
(n  =  20) 

Group 

Experienced  Group 
(n  =  20) 

Low  score 

14.0 

20.0 

High  score 

29.0 

32.0 

Mean 

21.5 

25.2 

Standard  deviation 
t  (df  =  38) 

4.2 

-3.0,  £<  .01 

3.5 

5 


The  difference  between  the  means  of  the  two  groups  is  3.7,  with  the  experienced 
group  performing  significantly  better  (£  <  .01).  Previous  computer  experience  does 
appear  to  be  a  central  factor  determining  successful  error  message  interpretation  even 
when  users  are  removed  from  the  actual  on-line  computer  task.  The  correlation  between 
user  profile  score  and  test  score  was  0.36  for  inexperienced  subjects  and  0.34  for 
experienced  subjects— both  nonsignificant  at  the  0.01  level. 

Of  interest  also  was  to  see  how  test  time  compared  between  the  inexperienced  and 
experienced  groups,  Table  2. 


Table  2 

Comparison  of  Inexperienced  and  Experienced 
Group  Test  Times  (Minutes) 


Inexperienced  Group 
(n  =  20) 

Experienced  Group 
(n  =  20) 

Lowest  time 

8.3 

7.4 

Highest  time 

29.5 

27.1 

Mean 

14.1 

14.1 

Standard  deviation 
t  (df  =  38) 

5.8 

0.00,  N.  S. 

5.8 

Because  of  their  previous  exposure  to  the  types  of  error  messages  employed  in  the 
test,  it  was  also  expected  that  the  Experienced  Group  would  be  able  to  complete  the  test 
items  faster.  However,  the  analysis  shows  the  means  and  standard  deviations  of  test 
times  for  the  two  groups  to  be  essentially  identical.  Thus  the  superior  performance  of  the 
Experienced  Group  can  not  be  attributed  to  its  spending  any  greater  time  on  the  error 
interpretation  task. 

Predictors  of  Success 


The  correlation  between  profile  score  and  test  score  for  all  40  subjects  was  0.53 
(p  <  .01).  A  regression  analysis  of  profile  subscores  to  predict  test  performance  revealed 
that  the  number  of  operating  systems  that  subjects  experienced  explained  as  much 
variance,  31  percent,  in  test  scores  as  did  the  entire  profile  score.  So,  successful 
interpretation  of  error  messages  was  more  a  function  of  the  number  of  operating  systems 
that  the  user  had  experienced  than  any  other  component  or  combination  of  components 
comprising  the  profile  score.  Item  score  and  time  correlation  was  only  0.34— so  the  more 
difficult  error  message  did  not  necessarily  take  more  time  to  interpret. 

Since  73.2  percent  of  the  error  messages  used  were  common  to  the  IBM  PC  system 
(Figure  1),  IBM  PC  users  (n  =  13)  were  compared  with  non-IBM  PC  users  (n  =  27).  IBM- 
experienced  users,  who  were  largely  a  subset  of  the  experienced  user  group  (n  =  20),  did 
better  than  non-IBM  users  in  interpreting  error  messages.  The  mean  test  score  for  IBM 
PC  users  was  25.9,  significantly  greater  than  the  22.1  of  non-IBM  users  (£<  .01).  Mean 
time  scores  were  also  similar:  13.2  minutes  for  IBM  users  compared  to  14.5  minutes  for  ail 
non-IBM  subjects,  and  14.1  for  both  experienced  and  inexperienced  user  groups  (Table  2). 
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Attributes  of  System  Error  Messages 


Listed  in  Table  3  are  those  messages  for  which  users  demonstrated  a  high  degree  of 
comprehension  and  those  for  which  they  did  not.  Nine  messages  are  listed  in  descending 
order  of  comprehension  for  each  category. 


Table  3 

The  Most  and  Least  Understood  Error  Messages 


Most  Understood 

Least  Understood 

(85-100  Percent  Comprehension) 

Question  Message 

Number 

(10- 

Question 

Number 

-25  Percent  Comprehension) 

Message 

2 

Insufficient  disk  space 

0  file(s)  copied 

10 

Illegal  entry 

13 

Try  again 

40 

Not  found 

23 

Sector  not  found 

22 

Read  fault 

36 

Incorrect  DOS  version 

16 

Too  many  parameters  entered 

39 

No  room  in  directory  for  file 

28 

Access  denied 

14 

Damaged  disk 

25 

Write  fault 

38 

Invalid  directory 

31 

Backup  file  sequence  error 

30 

Attempted  write  protect  violation 

26 

Write  protect 

15 

Disk  not  formatted 

20 

Invalid  drive  specification 

The  relationship  between  length  of  message  and  comprehension  is  not  clear.  The  41 
error  statements  varied  from  one  to  nine  words  with  a  mean  of  3.3.  The  mean  correct 
comprehension  rate  across  all  41  items  was  57  percent,  with  a  low  score  of  10  percent  for 
question  20  (invalid  drive  specification),  and  a  high  score  of  100  percent  for  question  2 
(insufficient  disk  space).  The  standard  deviation  of  the  percentage  scores  across  all  items 
was  27  percent.  The  least  understood  messages  were  successfully  interpreted  below  the 
chance  level  (25  percent).  However,  within  these  nine  items,  there  were  at  least  two 
other  alternatives  chosen  by  subjects.  Usually  one  highly  attractive,  but  incorrect 
alternative  misleads  most  users,  which  is  typical  of  multiple-choice  tests.  The  word  count 
of  the  most  understood  messages  averaged  more  than  three  words  and  for  the  least 
understood  message  less  than  three  words.  It  may  be  that  three  words  represent  the 
minimum  required  to  produce  an  understandable  prompt.  However,  correlation  between 
word  count  and  message  comprehension  for  all  41  error  statements  was  only  0.26. 

The  most  understood  messages  are  more  specific  in  terms  of  Shneiderman's  (19S2) 
dimension.  All  but  two  (questions  13  and  30)  refer  to  a  physical  entity  such  as  a  disk, 
directory,  file,  or  sector.  Interestingly  enough,  questions  26  and  30  resulted  in  extreme 
differences  in  interpretation,  although  the  only  difference  between  them  was  the  addition 
of  words  "attempted"  and  "violation"  in  question  30. 
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Both  the  most  and  least  understood  messages  suffer  from  lack  of  constructive 
guidance.  The  exception  is  question  13,  which  directs  the  user  to  take  further  action,  but 
without  specifying  a  cause  for  it.  Physical  format  was  constant  in  all  messages  and  the 
style  did  not  vary  noticeably  across  the  41  items.  Scapin  (1981)  found  computer-oriented 
commands  to  be  more  easily  understood  than  directions  that  used  ordinary  language. 
Shneiderman  (personal  communication,  1984)  reports  from  recent  studies  that  a  positive 
tone  is  ineffective  in  increasing  message  comprehension.  Some  respondents  performed 
better  to  prompts  that  were  stated  negatively.  However,  Shneiderman  advocates  using 
user-centered  language  that  allows  the  user  to  control  the  system  rather  than  the  system 
directing  what  the  user  does.  Prompts  should  avoid  the  imperative  forms,  such  as  ENTER 
DATA,  and  focus  on  user  control,  such  as  READY. 


DISCUSSION  AND  CONCLUSIONS 

The  population  of  computer  users  can  be  expected  to  grow  at  a  high  rate  for  the  next 
several  years.  However,  the  largest  growth  will  be  among  people  who  will  use  computers 
on  a  periodic  basis  (often  referred  to  as  discretionary  users).  These  are  the  people  who 
will  be  encountering  error  messages  most  frequently  because  of  limited  experience. 

The  results  of  this  study  indicate  that  people  do  learn  to  interpret  error  messages 
better  with  experience.  This  fact,  however,  is  of  little  comfort  to  the  discretionary  user 
who  will  use  the  system,  but  not  frequently  enough  to  gain  sufficient  experience  to 
become  proficient.  Additionally,  the  poor  level  of  performance  of  experienced  users 
indicates  that  the  interpretability  of  these  error  messages  is  still  far  below  what  should  be 
expected  for  a  system  that  is  to  serve  discretionary  users  as  well. 

On  the  average,  subjects  correctly  interpreted  23.3  error  messages  or  only  57 
percent.  Error  messages  need  to  be  much  more  understandable  for  users  to  perform 
efficiently  on  their  systems.  When  more  experienced  users  are  separated  out,  mean 
performance  increases  only  to  25.2  or  61  percent  correct  interpretation  of  error 
messages. 

The  difference  between  inexperienced  and  experienced  users  in  their  ability  to 
interpret  error  messages  is  significant,  which  further  highlights  the  inadequacy  of 
command  system  prompts  for  inexperienced  users  as  well  as  for  people  who  have  worked 
on  numerous  microcomputer  systems  in  a  variety  of  contexts.  For  them,  system  message 
interpretation  should  be  near  perfect.  Time  on  task  was  not  a  factor  in  explaining  the 
difference  between  experienced  and  inexperienced  subjects,  for  both  groups'  means  and 
standard  deviations  of  performance  time  were  practically  identical.  The  generalizability 
of  these  results  is  strengthened  by  the  subject  sample,  which  represents  a  wide  range  of 
ages  and  educational  levels.  It  included  civilian  and  military  personnel,  students,  clerical 
and  professional  people,  and  people  of  both  sexes. 

The  error  messages  were  for  tasks  related  to  both  device  and  data-handling  functions, 
tasks  well  within  the  realm  of  achievement  by  the  most  inexperienced  user.  The  error 
messages  themselves  are  commonly  used~73  percent  of  them  were  derived  from  software 
designed  for  the  most  popular  microcomputer  on  the  market  and  44  percent  from  software 
designed  for  a  microcomputer  used  most  frequently  by  the  military.  Users  of  the  most 
popular  system  fared  better  than  did  the  experienced  group  as  a  whole  and  their 
performance  was  significantly  better  than  that  of  people  inexperienced  on  that  system. 
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RECOMMENDATIONS 


The  need  to  improve  system  error  prompts  so  that  they  are  unambiguous  and 
comprehensible  is  of  high  priority.  Shneiderman  (1982)  suggests  principles  of  system 
message  design:  Messages  must  be  specific,  offer  constructive  guidance,  employ  user- 
centered  phrasing,  and  be  presented  in  an  appropriate  format.  Most  importantly,  a 
prompt  should  be  specific  and  prescriptive,  two  dimensions  that  offer  the  most  promise 
for  improving  user-computer  communication. 

Inexperienced  users  suffer  the  most  from  incomprehensible  feedback.  All  users  do 
poorly  when  the  context  of  the  task  and  the  situational  cues  of  the  on-line  operational 
environment  are  absent.  But  error  messages  should  not  be  dependent  on  contextual  cues, 
which  may  not  exist.  The  user  needs  to  be  told  exactly  what  he  or  she  has  done  wrong  and 
what  needs  to  be  done  to  recover  from  the  error  and  correct  the  situation. 

A  study  based  on  the  results  of  this  experiment  would  be  one  that  takes  the 
inadequate  error  messages  identified  here  and  redesigns  them,  demonstrating  their 
relative  effectiveness  through  replication.  This  could  be  done  by  rewriting  those 
messages  that  yielded  less  than  30  percent  correct  interpretation.  Those  items  that 
already  yield  a  high  percentage  of  correct  interpretations  could  be  used  as  controls  for 
test-retest  reliability.  Effective  messages  should  continue  to  be  effective,  and  redesigned 
messages  should  reflect  considerable  improvement  as  measured  by  scores  for  correct 
interpretation. 
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APPENDIX  A 

ERROR  MESSAGE  TEST  INSTRUCTIONS  AND  QUESTIONS 
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TEST  INSTRUCTIONS 


SCREEN 

1  Welcome  to  the  Zenith  microcomputer, 

What  is  your  name?  Joe  (blinking) 

Please  wait  for  the  next  message  to  appear  on  the  screen. 

2  Joe 

This  is  a  survey  to  gather  data  for  a  study  on  error  messages.  You  will  receive 
no  feedback  to  tell  you  if  your  response  to  a  particular  error  message  is  correct. 
Currently,  there  is  no  basis  for  comparison.  If  you  wish  to  know  the  result  of 
this  survey,  please  inform  the  person  in  charge  of  the  session  that  you  wish  to  be 
contacted  when  the  analysis  has  been  completed.  PLEASE  WAIT  FOR  A 
MESSAGE  TO  TELL  YOU  WHEN  TO  BEGIN. 

3  Joe,  please  type  in  your  assigned  ID  NUMBER. 

?  _1_  (blinking) 

Joe, 

A  series  of  error  messages  associated  with  the  ZENITH  will  be  displayed  on  the 
screen.  Each  error  message  will  have  4  responses.  Your  task  is  to  pick  the  one 
best  response  for  each  error  message. 

When  you  are  ready,  press  any  key  to  start. 


SCREEN 


4  Question  1  of  41 

VI  hich  is  the  best  interpretation  for  the  following  error  message? 

File  cannot  be  copied  onto  itself 
0  file(s)  copied 

a.  File  to  be  copied  does  not  exist 

b.  Error  in  copy  command 

c.  Second  disk  to  be  copied  or  not  specified 

d.  Insufficient  space  on  disk  to  make  copy 

Enter  your  answer  by  using  one  of  the  letters  (a  to  d)  and  then  press  -RETURN-. 
Answer _ (blinking) 


All  of  the  following  41  test  items  are  presented  with  screen  (4)  format. 
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ERROR  MESSAGE  TEST  QUESTIONS 


File  cannot  be  copied  onto  itself 
0  File(s)  copied 

a.  File  to  be  copied  does  not  exist 

b.  Error  in  copy  command 

*c.  Second  disk  to  be  copied  on  not  specified 
d  Insufficient  space  on  disk  to  make  copy 

Insufficient  disk  space 
0  File(s)  copied 

a.  Copy  command  error 

*b.  No  space  on  disk  to  make  copy 

c.  File  does  not  exist 

d.  Second  disk  not  specified 

File  not  found 
*a.  File  not  in  directory 

b.  Accessing  wrong  disk  drive 

c.  File  name  in  error 

d.  File  does  not  exist 

Bad  command  or  file  name 

a.  Erroneous  command  name 

b.  File  does  not  exist 

c.  File  not  in  directory 
*d.  Entry  error 

No  files 

*a.  File  does  not  exist 

b.  Wrong  disk  drive  specified 

c.  Insufficient  disk  space 

d.  File  extension  mission 

0  file(s)  copied 

a.  Insufficient  disk  space 
b-  File  cannot  be  copied  onto  itself 
*c.  Error  in  coDy  command 

d.  File(s)  not  in  directory 

Redo  from  start 

a.  Retype  directory  name 
*b.  Reenter  command 

c.  Change  disk  drives 

d.  Log  on  system  again 

Syntax  Error 

*a.  Error  in  command  format 

b.  Entry  misspelled 

c.  Error  in  entry  spacing 

d.  Too  many  parameters  entered 


9.  Entry  aborted 
*a.  Entry  ignored 

b.  Returned  to  system  mode 

c.  Entry  temporarily  stored  in  buffer 

d.  Illegal  command 

10.  Illegal  entry 

a.  Entry  misspelled 

b.  Error  in  entry  spacing 
*c.  Erroneous  entry 

d.  Syntax  error 

12.  Checksum  error 

a.  Checksum  file  does  not  exist 
*b.  Inaccurate  data  transmission 

c.  Recheck  all  of  preceding  checksum  data 

d.  Damaged  disk  storage 

13.  Try  again 

a.  Return  key  not  entered 
*b.  Reenter  command 

c.  File  not  on  disk 

d.  Processor  not  ready 

14.  Damaged  disk 

a.  Unformatted  disk 

b.  Not  a  Z-DOS  disk 

*c.  Cannot  read  to/write  from  disk 
d.  Wrong  disk  sector  size 

15.  Disk  not  formatted 

a.  Error  in  command  format 

b.  Damaged  disk 

*c.  Disk  not  prepared  for  use 
d.  Not  a  Z-DOS  disk 

16.  Too  many  parameters  entered 

a.  Too  many  spaces  entered 

*b.  Too  many  alphanumeric  characters  entered 

c.  Parameters  outside  of  acceptable  range 

d.  Syntax  error 

17.  Reference  made  to  a  nonexistent  file 
*a.  File  not  in  directory 

b.  Accessine  wrong  disk  drive 

c.  Erroneous  file  name 

d.  File  copied  onto  itself 

18.  Disk  error  reading 

*a.  Unable  to  read  first  disk  record 

b.  Damaged  disk 

c.  Unformatted  disk 

d.  Not  a  Z-DOS  disk 
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19. 

Disk  error  writing 

a. 

Disk  write  protected 

*b. 

Unable  to  write  first  disk  record 

c 

Accessing  wrong  disk  drive 

d. 

Unformatted  disk 

20. 

Invalid  drive  specification 

a. 

Specified  drive  not  turned  on 

b. 

Specified  drive  not  connected  to  computer 

c. 

Specified  drive  has  no  disk 

*d. 

Specified  drive  contains  unbootable  disk 

21. 

Non 

-DOS  disk 

a. 

Accessing  wrong  disk  drive 

*b. 

Unformatted  disk 

c. 

Current  operating  system  required  on  computer 

d. 

Command  error 

22. 

Read  fault 

a. 

Read  only  memory  (ROM)  full 

b 

Unformatted  disk 

c 

Current  operating  system  required  on  computer 

d. 

Command  error 

23. 

Sector  not  found 

*a. 

Data  sector  on  disk  cannot  be  located 

b. 

Unformatted  disk 

c. 

Disk  sector  damaged 

d. 

Sector  defined  by  incompatible  operating  system 

24. 

Seek 

a. 

File  extension  mission 

*b. 

Unable  to  locate  proper  disk  track 

c. 

Disk  sector  cannot  be  located 

d. 

Accessin6  wrong  disk  drive 

25. 

Write  fault 

a. 

Disk  write  protected 

*b. 

Retry 

c. 

Damaged  disk 

d. 

Abort 

26. 

Write  protect 

a* 

Disk  program  copyrighted 

b. 

Files  not  on  disk 

c. 

Disk  full 

*d. 

Files  cannot  be  detected 

27. 

About  to  generate.EXE  file 

Change  disks  (hit  ENTER) 

a. 

Accessing  wrong  disk  drive 

b. 

Unformatted  disk 

c. 

File  does  not  exist 

*d. 

Insert  program  disk 

28.  Access  denied 

a.  Security  violation 
*b.  Disk  write  protected 

c.  Disk  drive  not  turned  on 

d.  Proprietary  information 

29.  All  specified  file(s)  are  contiguous 

a.  Wrong  directory 

*b.  Files  written  sequentially  on  disk 

c.  Files  are  in  subdirectory 

d.  Files  are  separately  inaccessible 

30.  Attempted  write  protect  violation 
*a.  Cannot  write  on  disk 

b.  Disk  unformatted 

c.  File  cannot  be  printed 

d.  File  extension  needed  on  file  name 

31.  Backup  file  sequence  error 

a.  Backup  file  on  wrong  disk  drive 

b.  Damaged  disk  containing  file 

*c.  Disk  with  first  part  of  file  not  inserted 
d.  File  stored  on  disk  out  of  sequence 

32.  Bad  or  missing  Command  Interpreter 

*a.  COMMAND.COM  missing  from  DOS  disk 

b.  Damaged  system  disk 

c.  Unformatted  disk 

d.  DOS  system  disk  write  protected 

33.  Cannot  edit.BAK  file— rename  file 

a.  File  not  on  disk 

b.  Disk  write  protected 

c.  Accessing  wrong  disk  drive 
*d.  Cannot  change  backup  file 

34.  Disk  boot  failure 

a.  Disk  has  incompatible  operating  system 

b.  Accessing  wrong  disk  drive 

*c.  DOS  failed  to  load  into  memory 

d.  Hardware  malfunction 

35.  Divide  overflow 

*a.  A  program  tried  to  divide  by  zero 

b.  Insufficient  disk  space  to  store  answer 

c.  Variable  for  answer  not  defined  large  enough 

d.  Trying  to  divide  by  too  large  a  number 

36.  Incorrect  DOS  version 
a.  CPM  disk  used 

*b.  Wrong  version  of  DOS  used 

c.  Wrong  disk  drive  used 

d.  Disk  damaged 
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37.  Invalid  baud  rate  specified 

a.  Printer  buffer  overflow 

*b.  Computer  to  printer  data  transmission  rate  unsynchronized 

c.  Incorrect  port  connection  to  printer 

d.  File  too  large  to  be  printed 

38.  Invalid  directory 

a.  No  files  in  directory 

b.  Directory  is  full 

*c.  Directory  in  specified  path  does  not  exist 
d.  Directory  name  previously  used 

39.  No  room  in  directory  for  file 

a.  Directory  name  contains  too  many  characters 
*b.  Directory  on  specified  file  is  full 

c.  Extension  needed  for  file  name  in  specified  directory 

d.  Directory  reserving  space  for  a  subdirectory 

40.  Not  found 

a.  Disk  not  labeled 

b.  File  not  on  disk 

c.  Unable  to  access  specified  disk  drive 

*d.  String  specified  by  EDIT  command  not  found 

41.  Printer  error 

a.  Error  in  print  command 

b.  Printer  queue  is  full 

c.  File  not  saved  before  printing 
*d.  Printer  is  off-line 
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APPENDIX  B 
USER  QUESTIONNAIRE 
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USER  QUESTIONNAIRE 


User  No. 


Please  complete  the  following  questionnaire.  Your  answers  will  be  completely 
confidential.  The  responses  you  give  will  be  used  only  to  study  differences  between 
people  of  different  backgrounds  as  to  how  they  relate  to  knowledge  of  and  proficiency 
with  computers.  Check  the  appropriate  items  that  apply  to  you. 

Gender:  Male  (1) _  Female  (2) _ 

Age:  15-25  years  (1)  _ 

26-35  years (2)  _ 

36-45  years (3)  _ 

46-55  years  (4)  _ 

56-65  years (5)  _ 

66  and  over  (6)  _ 

Indicate  highest  general  education  level: 

Less  than  high  school  graduate/ equivalency  (1) _ 

High  school  graduate/equivalent  (2) _ 

Associate  Arts  Degree  (2  years  college)  (3) _ 

Bachelor's  Degree  (4) _ 

Master's  Degree  (5) _ _ 

Doctorate  Degree  (6)  _ _ 

Number  of  computer-related  (information  science,  data  analysis,  computer-assisted 
instruction  languages,  etc.)  courses  completed: 

(0)  None _ 

(1)  One  _ 

(2)  Two  -  four  _ 

(3)  Five  or  more _ 

Specialized  computer  training: 

(0) _ None  Number  of  hours 

used  last  month  = 


Programming 
languages  used  = 

Assembler  (1) 

Fortran/Cobal  (2) 

Basic  (3) 

APL  (4) 

PL1  (5) 

Pascal  (6) 

Ada  (7) 

"C"  (8) 

Others  _  (9) 
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Do  you  use  a  computer  at  home  and/or  at  work? 

(0)  No _  (1)  Yes _ 

What  percentage  of  your  working  time  is  spent  using  a  computer? 

(0)  None  _ 

(1)  0-10%  _ 

(2)  11-25%  _ 

(3) 26-50%  _ 

(4)  50%  _ 

For  what  purposes  do  you  use  a  computer? 

(0)  None 

(1)  Word  processing  _ 

(2)  Games  _ _ _ _ _ 

(3)  Programming _ 

(4)  Telecommunications/message  handling _ 

(5)  Educational/tutorials  _ _ 

(6)  Data  base  mgt./business  applications  _ 

(7)  Other _ _ _ 


How  many  word  processing  text  editors  have  you  used? 

(0)  None _ _ 

(1)  One -three  _ _ _ 

(2)  More  than  three  _ _ _ _ 

List  any  interactive  computer  system  (stand-along  or  timesharing)  you  have  used: 

(0)  None  _ 

(1)  _ 

(2)  _ 

(3)  _ 

(4)  _ 

(5)  _ 

(6)  _ 

What  operating  systems  have  you  used? 

(0)  None  _  / 

(1) CPM  _ 

(2)  Z-DOS  _ 

(3)  MS-DOS _ 

(4)  UNIX  _ 

(5)  Other  _ _ _ 
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DISTRIBUTION  LIST 


Military  Assistant  for  Training  and  Personnel  Technology  (OSUDS)  (R&AT) 

Chief  of  Naval  Operations  (OP-01B7)  (2),  (OP-14),  (OP-95),  (OP-987H) 

Director  of  Navy  Laboratories  (ONT  0722) 

Commander,  Naval  Civilian  Personnel  Command 

Commanding  Officer,  Service  School  Command,  San  Diego  (Code  3200) 

Commanding  Officer,  Naval  Health  Sciences  Education  and  Training  Command,  Bethesda, 
MD 

Chief  of  Naval  Research  (Code  100) 

Office  of  Naval  Research,  Detachment  Pasadena 
Chief  of  Naval  Education  and  Training  (Code  00) 

Director,  Naval  Education  and  Training  Program  Development  Center,  Millington,  TN 
Director,  Naval  Education  and  Training  Program  Development  Center,  Great  Lakes,  IL 
Commanding  Officer,  Naval  Education  and  Training  Program,  Pensacola,  FL  (Code  09), 
(Personnel  and  Training  Research) 

Commanding  Officer,  Naval  Education  and  Training  Support,  Atlantic  (Code  N3) 
Commanding  Officer,  Naval  Education  and  Training  Support  Center,  Pacific 
Education  Specialist,  VT-10,  Pensacola,  FL 
Chief  of  Naval  Technical  Training  (Code  00) 

Commanding  Officer,  Naval  Technical  Training  Center,  Corry  Station,  Pensacola,  FL 
(Code  1 0 1 B) 

Commander,  Training  Command,  U.S.  Atlantic  Fleet  (Code  01A) 

Commanding  Officer,  Fleet  Combat  Training  Center,  Pacific 
Commanding  Officer,  Fleet  Training  Center,  San  Diego 
Commander,  Fleet  Training  Group,  Pearl  Harbor 
Commander,  Naval  Ocean  Systems  Center 
Commander,  Naval  Weapons  Center 

Commander,  David  W.  Taylor  Naval  Ship  Research  and  Development  Center 
Commanding  Officer,  Naval  Training  Equipment  Center  (Technical  Library)  (5),  (Code  10), 
(Code  7),  (Code  7 A) 

Commander,  Naval  Sea  Systems  Command 

Commander,  Space  and  Naval  Warfare  System  Command  (SPA WAR-06/ AS W-50) 

Mr.  Greenup,  Education  Center  MCDEC,  Quantico,  VA 
Chief,  ARI-USAREUR  (Library)  (2) 

Program  Manager,  Life  Sciences  Directorate,  Bolling  Air  Force  Base  (AFOSR/NL) 
Commander,  Air  Force  Human  Resources  Laboratory,  Brooks  Air  Force  3ase  (AFHRL/MO 
Manpower  and  Personnel  Division),  (Scientific  and  Technical  Information  Officer), 
(TSRL/Technical  Library  FL  2870) 

Commander,  OPSTNGDIV  Air  Force  Human  Resources  Laboratory,  Williams  Air  Force 
Base  (AFHRL/OT) 

Commander,  Air  Force  Human  Resources  Laboratory,  Wright-Patterson  Air  Force  Base 
(AFHRL/LR-TDC) 

Commander,  Air  Force  Human  Resources  Laboratory,  Lowry  Air  Force  Base  (AFHRL/ID) 
Director,  Training  Systems  Development,  Hq  ATC/XPTD,  Randolph  Air  Force  Base 
Director,  AFLMC/XRU,  Technical  Reference  Library,  Gunter  AFS,  AL 
Superintendent,  Naval  Postgraduate  School 
Director  of  Research,  U.S.  Naval  Academy 
Center  for  Naval  Analyses 

Dr.  Ackles,  Canadian  Defense  Liaison  Staff,  Washington,  DC 
Canadian  Forces  Personnel,  Applied  Research  Unit,  Canada 
DPSRSC,  National  Defense  Headquarters,  Canada 
Ministry  of  Defense,  Senior  Psychologist,  Naval,  England 


D.  Dennison,  Army  Personnel  Research  Establishment,  Personnel  Psychological  Division, 
England  (2) 

Science  3  (RAF),  Lacon  House,  England 
1  Psychological  Research  Unit,  NBH  3-44,  Australia 

Directorate  of  Psychology-AF,  Department  of  Defense  (Air  for  CE),  Australia 
Navy  Psychology,  Australia  (2) 

Defense  Psychology  Unit,  Defense  HQ,  New  Zealand  (2) 

Defense  Technical  Information  Center  (DDAC)  (2) 
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